@nicholasjackson couple updates to talk through:

• Introduce the idea of policy across other things (http, grpc, tcp).
• Get rid of the many -> many relationship in TrafficRole with rules.
• Move TrafficRoleBinding to only supporting ServiceAccount.
• Added a section for admission control and RBAC. I'm planning on fleshing that out with some OPA configurations that show controlling service accounts on pods and users creating arbitrary role/rolebindings.

Outstanding issues with this current proposal:

• subjects on TrafficRole doesn't feel right and in this current proposal does not allow selecting individual routes.
• Still feels like there needs to be a ClusterTrafficRole/ClusterTrafficRoleBinding for super services (ones in kube-system for example).

There's a couple different ways this all can come together and I'm looking forward to some brainstorming tomorrow morning!

@grampelberg posting this here rather than sending a direct email but here are some thoughts and explanation on Envoy topology. Sorry if this is a little rough, I am multi-tasking while eating dinner.

Envoy defines listeners, these generally have two uses:

• Ingress flow into the data plane

• Egress expose a local reference to an upstream service

• Listeners have routes, the route is generally L7 specific, tho it is possible to configure basic L4

• HTTP Routes have matchers which can have a path and method, each route references a Cluster, a Cluster is a *collection of endpoints which are either defined statically or from a Service Catalog.

In terms of object mapping Envoy to HTTPService(Route):
HTTPService or gRPCService would be an Envoy Listener
Routes are routes
Service would be an Envoy Cluster

This is not 100% exact but it would be pretty easy to map an HTTPService to Envoy configuration objects

kind: HTTPService // Envoy Listener
apiVersion: v1beta1
metadata:
  name: foo
  namespace: default
resources:
  # v1.ObjectReference
  - kind: Service
    name: foo //Envoy cluster
routes:
  - name: admin // Envoy route
    methods: // Envoy route match
    - GET
    pathRegex: "/admin/.*" // Envoy route match
    - name: default // Envoy route
      methods: ["*"]
      pathRegex: ".*"

With this logical model you could move service to route, it might also make sense to reason about HTTPService, etc as HTTPListener, gRPCListener, TCPListener.

kind: HTTPListener
apiVersion: v1beta1
metadata:
  name: foo
  namespace: default
routes:
  - name: admin
    service: foo
    methods:
    - GET
    pathRegex: "/admin/.*"
    - name: default
      methods: ["*"]
      pathRegex: ".*"

Listener or whatever the term is would be a higher level object, not owned by policy but the configuration for the data plane which is yet to be defined. It would be accessed from a TrafficRole like so:

kind: TrafficRole
apiVersion: v1beta1
metadata:
  name: path-specific
  namespace: default
resource:
  name: foo
  kind: Deployment
subjects:
  - kind: HTTPListener
    name: admin
    routes:
    - admin
    - default

The tradeoff with this approach is that Role would not be able to sub filter a route, i.e. limit a route to a particular subpath or method. This would be achieved by adding an explicitly named route on the higher level listener.

The example below defines an Envoy HTTP based public listener, this will be the mTLS ingress to the data plane, generally, this points to a local service.

          {
          "@type": "type.googleapis.com/envoy.api.v2.Listener",
          "name": "public_listener:${POD_IP}:20000",
          "address": {
            "socketAddress": {
              "address": "${POD_IP}",
              "portValue": 20000
            }
          },
          "filterChains": [
            {
              "filters": [
                {
                  "name": "envoy.http_connection_manager",
                  "config": {
                    "stat_prefix": "ingress_http",
                    "route_config": {
                      "name": "local_route",
                      "virtual_hosts": [
                        {
                          "name": "backend",
                          "domains": ["*"],
                          "routes": [
                            {
                              "match": {
                                "prefix": "/"
                              },
                              "route": {
                                "cluster": "local_app"
                              }
                            }
                          ]
                        }
                      ]
                    },
                    "http_filters": [
                      {
                        "name": "envoy.router",
                        "config": {}
                      }
                    ]
                  }
                }
              ]
            }
          ]
        }

Example cluster definition for a local app (data plane ingress)

              {
                "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                "name": "local_app",
                "connect_timeout": "5s",
                "http2_protocol_options": {},
                "hosts": [
                 {
                  "socket_address": {
                   "address": "127.0.0.1",
                   "port_value": 9090
                  }
                 }
                ]
              }

Example cluster definition using endpoint discovery service (typically remote service catalog)

              {
                "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                "name": "service:emojify-cache",
                "type": "EDS",
                "http2_protocol_options": {},
                "eds_cluster_config": {
                  "eds_config": {
                    "ads": {}
                  }
                },
                "connect_timeout": "5s" // connection to endpoint not request, request timeout is on route
              }

Alright, new update:

• Change policy name to access control. There's a bunch of different policies that will be valid (retries, timeouts, rate limiting, ...).
• Remove the binding from HTTPService (and rename to HTTPRoute). Following the thoughts behind having multiple different policies, it makes sense for a policy to bind resources and routes.
• Describe sub-selection. HTTPRoutes will rarely be used as a single object, explain how to select an individual item from that bucket.
• Explain how unauthenticated clients will get roles bound to them.
• Enumerate the prometheus metrics and k8s liveness/readiness use cases.

@nicholasjackson I think I covered everything we chatted about this morning. Let's cover in the call tomorrow.

@grampelberg Awesome, I think this is looking pretty good to merge.

I think there is one addition I need for my implementation but let's get this merged I can raise a PR separately for this.

@nicholasjackson there's definitely some more things I want to add as well. I'm into the merge and iterate idea. Let's make sure everyone's happy with the renaming and then merge later today.

I'm interested in what else you'll need =)

Reached consensus on merging this out of band.